Skip to content

feat(autoops_es): add ingest volume enrichment fields - #52226

Open
ilya-shevelyov wants to merge 3 commits into
mainfrom
51404-added-ingest-volume-enriched-fields
Open

feat(autoops_es): add ingest volume enrichment fields#52226
ilya-shevelyov wants to merge 3 commits into
mainfrom
51404-added-ingest-volume-enriched-fields

Conversation

@ilya-shevelyov

@ilya-shevelyov ilya-shevelyov commented Jul 23, 2026

Copy link
Copy Markdown

Closes #51404

Summary

Implements #51404 — extends the autoops_es.cat_shards and autoops_es.node_stats metricsets with ingest/bulk snapshot fields and per-second rate enrichers for upcoming resource-utilization insights.

Part A — cat_shardsnode_index_shards

  • Appends btsi, bto, dataset, dvc columns to the _cat/shards h= request (ES 8.0+)
  • Aggregates them as primary-shard-only snapshot fields: bulk_total_size_in_bytes, bulk_total_operations, total_data_set_size_in_bytes, dense_vector_count
  • Adds 4 rate enrichers: ingest_docs_per_second, ingest_bytes_per_second, bulk_bytes_per_second, bulk_operations_per_second

Part B — node_stats

  • Parses indices.bulk from _nodes/stats response (DictOptional — gracefully absent on ES 7.x): bulk_total_size_in_bytes, bulk_total_operations
  • Adds the same 4 rate enrichers backed by indices.docs.count, indices.store.size_in_bytes, and indices.bulk.*

Test plan

  • cat_shards/cache_test.go — 4 new rate enricher tests (no-cache, with-cache, no-change, new-node)
  • cat_shards/index_shards_test.go — primary-only aggregation for new snapshot fields
  • cat_shards/data_test.go — nil rate assertions on replica-only node
  • cat_shards fixture (8.15.3) updated with bulk/dataset/dense-vector columns
  • node_stats/cache_test.go — new rate enrichers covered across all existing cache test scenarios
  • node_stats/data_test.go — snapshot field assertions from fixtures; nil/NotNil rate assertions per version (bulk rates nil on 7.x — no indices.bulk in that ES version)
  • node_stats fixture (8.15.3, 9.2.0) already contained indices.bulk; 7.17.0 left unchanged (field absent in that ES version)

🤖 Generated with Claude Code

Extends cat_shards and node_stats metricsets with bulk/dataset/dense-vector
snapshot fields and ingest/bulk rate enrichers for resource-utilization insights.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ilya-shevelyov
ilya-shevelyov requested a review from a team as a code owner July 23, 2026 10:49
@botelastic botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
  • /test : Run the Buildkite pipeline.

@mergify

mergify Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ilya-shevelyov? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@consulthys consulthys left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!!
Just a couple renames

Comment thread x-pack/metricbeat/module/autoops_es/cat_shards/data.go Outdated
Comment thread x-pack/metricbeat/module/autoops_es/cat_shards/deserialize.go Outdated
@pickypg pickypg added Team:Opex backport-9.4 backport-9.5 Automated backport to the 9.5 branch labels Jul 23, 2026
@botelastic botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 23, 2026
@ilya-shevelyov
ilya-shevelyov force-pushed the 51404-added-ingest-volume-enriched-fields branch from a9b5d70 to 2024378 Compare July 23, 2026 14:50
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ilya-shevelyov
ilya-shevelyov force-pushed the 51404-added-ingest-volume-enriched-fields branch from 2024378 to d82ae6e Compare July 23, 2026 15:06
@ilya-shevelyov ilya-shevelyov added backport Module:AutoOps ES AutoOps Elasticsearch module labels Jul 23, 2026
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport backport-9.4 backport-9.5 Automated backport to the 9.5 branch Module:AutoOps ES AutoOps Elasticsearch module Team:Opex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add ingest volume enrichment fields

3 participants